A Multimodal Communication Interface for Assistive Robots for the Elderly

The RoboHelper project aims at building an effective and user friendly communication interface for assistive robots that can help the elderly live independently at home. Such communication interface should incorporate multiple modalities of communication, since collaborative task-oriented human-human communication is inherently multimodal. Data was collected from twenty collaborative task-oriented human-human communication sessions between a helper and an elderly person in a realistic setting (fully functional studio apartment). We observed that these interactions involve a specific type of gestures, Haptic-Ostensive (H-O) actions [1]. H-O actions manipulate objects, but they also often perform a referring function. For example, the helper may open a drawer without saying a word while looking for a pot, and the elder may say “Not there”. “there” is a socalled referring expression whose referent (the drawer) was established in the context without any words, but through the action of opening the drawer itself, namely via visual and/or haptic cues (note that as far as the helper is concerned, even if they were blindfolded they could establish the existence of the drawer via the action they perform). We ultimately envision an interface that can exploit a multiplicity of cues – spoken, visual and haptic – to support communication. In our project, we investigate which roles H-O actions play in interaction, since H-O actions have not been studied as much as spoken and visual signals; and we show that we can actually recognize H-O actions from the haptic signals. We believe that our work on H-O actions is applicable to any domain where participants (including robots) collaborate on tasks that involve object manipulation.